Multi-modal interaction on multi-touch display

ABSTRACT

Embodiments are disclosed herein that relate to multi-modal interaction on a computing device comprising a multi-touch display. One disclosed embodiment comprises a method of multi-modal interaction including recognizing a hand posture of a user&#39;s first hand directed at the display and displaying a modal region based on the hand posture, wherein the modal region defines an area on the display. The method further includes receiving an input selecting a mode to be applied to the modal region, wherein the mode indicates functionalities to be associated with the modal region and defines a mapping of touch gestures to actions associated with the mode. The method further includes, while the modal region remains displayed, recognizing a touch gesture from a user&#39;s second hand directed at the display within the modal region and performing an action on the display based upon a mapping of the touch gesture.

BACKGROUND

Computing systems having touch-gesture recognition capabilities may beconfigured to detect input touches from a user, such as touch inputs onthe display surface, near-touch gestures directed toward the surface,etc. For the case of computing systems having multi-touch displays,multi-touch gestures from a single user may be detected, as well asmultiple touch inputs (including multiple multi-touch inputs) fromvarious users, and touch gestures in which an input is made via movementof a touch on or over the display.

Different gestures made on a multi-touch display may be mapped todifferent user interface controls. If each control is mapped to adifferent gesture, then a large set of user interface functionalitiesmay correspond to a large set of touch gestures. However, in practice, aset of physical gestures that a user may reasonably be able to performconsistently and reliably may be relatively small. To address thisissue, a user interface may utilize dedicated on-screen areas, such astouch buttons, sliders, checkboxes, and the like, in place of somegestures so that a total set of gestures utilized by a user may bereduced. However, such elements may clutter the user interface and leavelittle display space dedicated to actual content.

SUMMARY

Various embodiments that relate to multi-modal interaction on amulti-touch display are disclosed herein. For example, one disclosedembodiment provides, on a computing system comprising a multi-touchdisplay, a method of multi-modal interaction. The method includesrecognizing a hand posture of a user's first hand directed at thedisplay and displaying on the display a modal region based on the handposture, wherein the modal region defines an area on the display. Themethod further includes receiving an input selecting a mode to beapplied to the modal region, wherein the mode indicates a family offunctionalities to be associated with the modal region and defines amapping of one or more touch gestures to one or more correspondingactions associated with the mode. The method further includes, while themodal region remains displayed, recognizing a touch gesture from auser's second hand directed at the display within the modal region onthe display, and performing an action on the display based upon amapping of the touch gesture from the user's second hand to the action.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically shows an example use scenario of multi-modalinteraction on a multi-touch display in accordance with an embodiment ofthe present disclosure.

FIG. 2 shows a flow diagram of an embodiment of an example method ofmulti-modal interaction.

FIG. 3 schematically shows an example time sequence of invoking anexample embodiment of a modal region displayed on a multi-touch display.

FIG. 4 schematically shows an example embodiment of a modal regiondisplayed on a multi-touch display.

FIG. 5 schematically shows an example of adjusting a location of a modalregion displayed on a multi-touch display.

FIG. 6 schematically shows an example of using a modal region inaccordance with an embodiment of the present disclosure.

FIG. 7 schematically shows an example time sequence of an embodiment ofa method of locking an embodiment of a modal region displayed on amulti-touch display.

FIG. 8 schematically shows an embodiment of a mode selection for a modalregion.

FIG. 9 schematically shows an example time sequence depicting anembodiment of a method of using a modal region.

FIG. 10 schematically shows another example time sequence depicting anembodiment of a method of using a modal region.

FIG. 11 schematically shows yet another example time sequence depictingan embodiment of a method of using a modal region.

DETAILED DESCRIPTION

Embodiments are disclosed herein that relate to enabling one or moreusers to interact with a multi-touch display via touch gestures, handpostures, etc. in a multi-modal and/or multi-user environment. Forexample, in some disclosed embodiments, a user may perform a handposture with one hand directed toward the display to bring up a modalregion within which the user's other hand may perform touch gestures,select settings from a contextual menu, etc. The position of such amodal region on the display may not only track the position of theuser's hand, but may be easily invoked, locked, or removed from thedisplay to facilitate a favorable user experience. Furthermore, multipleusers using a same display device, such as a tabletop surface computingsystem, may be able to each invoke their own modal region, and may eachwork in a different mode or in a same mode, facilitating a moreefficient group workspace.

As an example FIG. 1 illustrates an example use scenario 100 ofmulti-modal interaction on a computing device 101 comprising amulti-touch display 102, as viewed from overhead. At 104, a first user106 uses a modal region to manipulate content displayed on display 102.At 108, a second user 110 uses another modal region to manipulatecontent displayed on display 102. The computing device further comprisesa processor 112, and memory 114 containing instructions executable bythe processor to perform various functions, examples of which aredescribed herein, related to enabling such users to interact with thecomputing device. It can be appreciated that the example shown in FIG. 1is nonlimiting. As described below in more detail below, a variety ofdifferent types of multi-modal interaction may be used without departingfrom the scope of this disclosure.

FIG. 2 illustrates a flow diagram of an embodiment of an example method200 of multi-modal interaction, to be performed on a computing systemcomprising a multi-touch display. For example, the computing system mayhave a processor and memory comprising instructions stored thereon thatare executable by the processor to perform such a method. At 202, method200 includes recognizing a hand posture of a user's first hand directedat the display (e.g., touching the display). Such a hand posture may beany suitable hand posture of which the computing system is configured torecognize. Next, at 204, method 200 includes displaying on the display amodal region based on the hand posture. As such, the modal region may bedescribed as being “kinesthetically defined” since it is based on a handposture, as opposed to being, for example, temporally defined. The modalregion defines an area on the display. Additionally, the modal regionmay be displayed at a location on the display based on a location of thehand posture. For example, the modal region may be displayed in a closeproximity to the hand posture.

It can be appreciated that recognizing a hand posture is just onesuitable way of invoking a modal region. As another example, a user mayuse tools instead of their hands to either invoke the modal region orperform actions. For example, in some embodiments, a tagged objectrecognizable by the computing system or other suitable object may beused to invoke the modal region.

As an example, FIG. 3 illustrates a time sequence 300 that illustratesan invoking of an embodiment of a modal region on a multi-touch display.At a first time t₁, a document 302 having content 304 is displayed on amulti-touch display. At subsequent time t₂, a user's hand 306 isdirected toward the multi-touch display, as indicated by arrow 308. Asshown, the user's hand 306 is in a hand posture that is substantiallyL-shaped, wherein the user's thumb is extended outward from the hand. Itwill be appreciated that such a hand posture is nonlimiting, and is oneexample of a variety of possible hand gestures that may be used toinvoke a modal region. In response to recognizing the hand gesture, atsubsequent time t₃, a modal region 310 is displayed on the display. Inthis example, modal region 310 defines an area 312 on the display. Itwill be appreciated that a shape of area 312 is nonlimiting, and is oneexample shape of a variety of possible shapes a portion of the displayarea enclosed by a modal region may have. As another example, FIG. 4schematically shows an example of another embodiment of a modal region400 displayed on a multi-touch display, defining an area 402 of thedisplay.

The computing system may be further configured to display on themulti-touch display the modal region as tracking the hand posture whenthe hand posture changes its location on the display. In other words,the computing system may be configured to detect a change in thelocation of the hand posture and in response, adjust the location of themodal region. Accordingly, continuing with FIG. 2, method 200 mayfurther include detecting a change in the location of the hand posture,and in response, adjusting the location of the modal region. As anexample, FIG. 5 shows an example multi-touch display 500 displaying adocument 502, content 504 and photos 506. A user 508 may originallyinvoke a modal region 510 to modify content 504, as indicated in dashedline at 512. At a later time, the user may decide to work with photos506, and may adjust the location of modal region 510 simply by dragginga hand to a new location on display 500, as indicated in solid line at514.

Continuing with FIG. 2, at 205, method 200 may include assigning adefault mode to the modal region. The mode indicates a family offunctionalities to be associated with the modal region, as explained inmore detail below. Examples of possible modes include but are notlimited to word-processing, graphics editing, photography editing,spreadsheet-processing, etc. In other words, a mode may be virtually anygrouping of features a user may traditionally see on a traditional menusystem, feature list, ribbon, etc. of traditional applications. It willbe understood that, in other embodiments, no default mode may beassigned.

At 206, method 200 optionally includes receiving an input selecting amode to be applied to the modal region. Such a mode may, for example,override any default mode assigned at 205. The mode may be selected inany suitable manner. In some embodiments, a user may select the modefrom a list of modes displayed on the display via a touch gesture. Asanother example, a user may select the mode via a touch input directedat a virtual button displayed on the display. As yet another example, alist of available modes may be displayed on the display, and the usermay select a mode from the list via a touch input from the user's secondhand. In other embodiments, a user may select the mode from a virtualdial of modes displayed on the display, wherein a rotation gesture ofthe user's first hand virtually rotates the dial to select a mode. Inyet other embodiments, the hand posture used to invoke the modal regionmay further select a mode. For example, one hand posture may invoke apaintbrush mode while another hand posture may invoke an image editingmode.

Such a mode defines a mapping of one or more touch gestures to one ormore corresponding actions associated with the mode. For example, in aword-processing mode a pinch gesture may be mapped to an action thatdecreases font size, however in an image editing mode a pinch gesturemay be mapped to adjusting a size of an image. As another example, in aspreadsheet mode a finger touch may be mapped to selecting a cell,however in a photography mode a finger touch may be mapped to selectinga photograph. By mapping touch gestures to actions associated with themode, a user may utilize a smaller set of gestures since each gesturemay be “reused” in a different context based on the mode.

At 208, method 200 includes, while the modal region remains displayed,recognizing a touch gesture from a user's second hand directed at thedisplay within the modal region on the display. In some embodiments, themodal region remains displayed if the user's first hand is stillperforming the hand posture (e.g., still touching the display). However,in some embodiments, the modal region remains displayed if the user has“locked” the modal region in place, as will be discussed in more detailhereafter. With regard to the touch gesture, any suitable touch gesturethat the computing system is configured to recognize may be used.Nonlimiting examples of such a touch gesture include a finger tap, adouble tap, a finger drag, a holding gesture, a flicking gesture, adrag-and-drop gesture, etc. Such a touch gesture may also be amulti-touch gesture, wherein possible examples include but are notlimited to a pinch gesture, a reverse-pinch gesture, a two-finger tap, arotation gesture, etc.

In some embodiments of method 200, the user's first hand used to invokethe modal region may be a user's non-dominant hand, and the user'ssecond hand performing the touch gesture may be the user's dominanthand. For example, a right-handed user may use her left hand to invokethe modal region, and may then use her right hand to direct touchgestures at the area of the display defined by the modal region. Inother embodiments, a user may use a dominant hand to invoke a modalregion. Further, as described above, it can be appreciated that a usermay use tools instead of their hands when working with the modal region.For example, a pen or other suitable object may be used to perform amoded action in the modal region.

Next, at 210, method 200 includes performing an action on the displaybased upon a mapping of the touch gesture from the user's second hand tothe action. For example, in some embodiments, performing the action mayinclude changing a property of content displayed on the display. A smallnumber of more specific examples of such an action include, but are notlimited to, editing text, painting an image, adjusting photos in alayout, applying a sepia tone to a photo, opening an email application,or any other suitable action.

A modal region as described above may be described as operating in alayer “above” an application running on the computing system such thatit is used to modify content displayed “underneath” the model region.Thus, whereas traditionally a user may have to open and switch betweenseveral applications to perform a variety of computing tasks, the modalregion allows a user an easily-accessible multi-modal workspace in whichto efficiently perform such tasks.

The action performed on the display may be used to modify content insideand/or outside of the modal region. For example, in some embodiments,content inside a modal region may be modified. As an example, FIG. 6shows a possible use scenario of a user 600 editing a document 602 in anexample mode “Mode A.” As shown, content 604 within the modal region 606becomes activated and may be modified by actions based on touch gesturesreceived within the modal region. Accordingly, content 608 outside ofthe modal region may not be modified by such touch gestures within modalregion 606.

In other embodiments, the action performed on the display may modifycontent outside of the modal region. As an example, a gesture may beperformed within the modal region that adjusts a size of all imagesdisplayed on the display regardless as to their location and/orproximity to the modal region. Further, in some embodiments, a user maystart manipulating content inside the modal region, and during that sameinteraction (i.e., without lifting the touch gesture from the secondhand) end the touch gesture outside of the modal region, while in thesame mode. In such embodiments, a mode can then stay “persistent”outside of the modal region if the touch gesture was initiated inside ofthe modal region. It will be understood that whether a gesture modifiescontent inside or outside of the modal region may depend upon the modeapplied to the modal region.

In some embodiments, the computing system may be configured to recognizea plurality of hand postures of one or more other users, and display amodal region associated with each user's hand posture. As such, method200 may further include recognizing one or more hand postures, anddisplaying on the display a modal region for each user based on thelocation of the recognized hand posture. Since each user may utilizetheir own modal region of the multi-touch display, several users maywork on a single multi-touch display, thereby facilitating multi-userenvironments. Users may be working with the same content on the display,or different content. Thus, in some cases a user may select a same modefor their modal region as other users may be using. As an example, ateam of users may each utilize their modal region in a photo-editingmode, and work together to edit a set of photos displayed on themulti-touch display. Alternatively, a user may select a different modefor their modal region. As an example, one user may utilize their modalregion in a word-processing mode to add text to a document, whileanother user utilizes their modal region in an image editing mode toformat graphics of the same document.

Modal regions may be temporally overlapping, in that more than one modalregion may be utilized on the multi-touch display at one time. Further,in some embodiments, at some times, two or modal regions may be locatedin positions that may cause the modal regions to be spatiallyoverlapping. As such, the computing device may be further configured todetect “collisions” of modal regions, and deny or allow overlapping ofmodal regions based on the modes of the modal regions. For example, thecomputing system may, upon detecting a potential overlap of two modalregions displayed on the display, determine whether displaying the twomodal regions as overlapping is allowed or unallowed. Such a comparisonmay be based upon each mode active in each of the two regions. Forexample, the computing system may have a predetermined set of rulesallowing overlap for certain pairings of compatible modes, while notallowing overlap for other pairings. Upon making such a

Method 200 may further include detecting a lifting of the user's firsthand, and in response, ceasing display of the modal region. In otherwords, the modal region may be removed from the display by a user simplylifting their first hand from the display, thus removing the handposture from the display. As such, the user may then subsequently invokea modal region by performing a subsequent hand posture on the display.Thus, the modal region may be easily invoked and removed from thedisplay. In other embodiments, the modal region may persist for somepredetermined period of time after lifting of the first hand isdetected.

In some embodiments, a user may have the ability to lock their modalregion at a particular location on the multi-touch display, such thatthe user may then lift the first hand (i.e. the hand that invoked themodal region) from the display. In other words, in response to receivinga locking input, method 200 may include continuing to display the modalregion after the user's first hand is lifted. The modal region may belocked in any suitable manner. As an example, a user may select to lockthe modal region by a touch input directed at a virtual button displayedon the display. As another example, a user may perform a gesture ormodify their hand posture to lock the modal region.

FIG. 7 shows a time sequence illustrating an embodiment of locking amodal region. At an earlier time, a user's first hand 700 has invoked amodal region 702 displayed on the display. At a subsequent time t₁, theuser modifies his hand gesture by bringing his extended thumb toward hispalm as indicated at arrow 704. The computing system recognizes thisgesture as locking the modal region, allowing the user to lift his handsfrom the display and the computing system will continue to display themodal region. Accordingly, at subsequent times t₂ and/or t₃, the usermay utilize the locked modal region 702 with his first hand 700 and/orsecond hand 706 without having to maintain the hand posture on thedisplay.

It will be appreciated that a modal region may be configured tofacilitate user input in any suitable manner, various non-limitingexamples of which are described in more detail below.

FIG. 8 illustrates an example of mode selection for an embodiment of amodal region. The depicted example shows a modal region 800 invoked by ahand posture of the user's first hand 802. Upon invoking the modalregion, a user's second hand 804 may direct a touch input toward thedisplay within the modal region, upon which a menu 806 of possible modesmay be displayed. For the case of the nonlimiting example illustrated inFIG. 8, such a menu may include a photo layout mode, a help mode, asearch mode, a drawing mode, etc.

FIG. 9 illustrates an example time sequence of using an embodiment of amodal region to perform an action on an object displayed in the modalregion. The depicted example shows a time sequence, wherein a modalregion 900 has been invoked by a hand posture of the user's first hand902. Upon invoking the modal region, a user's second hand 904 may directa touch input toward an object 906 displayed on the display within themodal region 900, such as shown at time t₁. Upon recognizing the touchinput, at subsequent time t₂, a menu 908 is displayed which hasselectable options A, B, C, etc. For example, object 906 may be animage, and the menu options may include emailing the image, cutting theimage, copying the image, etc. At subsequent time t₃, the user maychoose to, for example, email object 906 by dragging object 906 to thecorresponding heading. In response to recognizing the dragging touchgesture of the user's second hand 904, the computing system may displayan action of displaying the user's email client. As such, the user mayeasily switch from manipulating the object in, for example, an imageeditor to emailing the image to a friend.

FIG. 10 illustrates another example time sequence of using an embodimentof a modal region to perform an action on an object displayed in themodal region, in the context of a photo editing mode. The depictedexample shows a time sequence, wherein a modal region 1000 has beeninvoked by a hand posture of the user's first hand 1002. Upon invokingthe modal region, a user's second hand 1004 may direct a touch inputtoward a photo 1006 displayed on the display within the modal region1000, such as shown at time t₁. In such an example, the modal region isin an example mode D. Upon recognizing the touch input, the multi-touchdisplay displays a menu 1008 of example options a, b, c and dcorresponding to actions that may be performed on the object 906, asshown at subsequent time t₂. Such actions may be based on the particulartype of photo editing mode selected for the modal region 1000, namelymode D. For example, mode D may be a filter mode of the photo editingapplication, and options a through d may represent various types ofpost-processing photographic digital filters, such as a sepia filter, ablack-and-white filter, etc. Thus, a user may drag photo 1006 to optiona corresponding to a sepia filter to apply a sepia tone to photo 1006.

FIG. 11 shows yet another example time sequence of using a modal regionin the context of a photo editing application as illustrated in FIG. 10.In this example, the user has selected a different mode for modal region1000, namely mode B. Upon invoking the modal region, a user's secondhand 1004 may direct a pinching touch gesture toward photo 1006, such asshown at time t₁. In such an example, the computing system may determinethat the pinching gesture made by the user corresponds to an action ofdecreasing the size of the photo 1006. Accordingly, at subsequent timet₂, the computing device performs the action of decreasing the size ofphoto 1006.

It will be appreciated that the computing devices described herein maybe any suitable computing device configured to execute the programsdescribed herein. For example, the computing devices may be a mainframecomputer, personal computer, laptop computer, surface computer, portabledata assistant (PDA), computer-enabled wireless telephone, networkedcomputing device, or other suitable computing device or devices, and maybe connected to each other via computer networks, such as the Internet.Such computing devices typically include a processor and associatedvolatile and non-volatile memory, and are configured to execute programsstored in non-volatile memory using portions of volatile memory and theprocessor. As used herein, the term “program” refers to software orfirmware components that may be executed by, or utilized by, one or morecomputing devices described herein, and is meant to encompass individualor groups of executable files, data files, libraries, drivers, scripts,database records, etc. It will be appreciated that computer-readablemedia may be provided having program instructions stored thereon, whichupon execution by a computing device, cause the computing device toexecute the methods described above and cause operation of the systemsdescribed above.

It should be understood that the embodiments herein are illustrative andnot restrictive, since the scope of the invention is defined by theappended claims rather than by the description preceding them, and allchanges that fall within metes and bounds of the claims, or equivalenceof such metes and bounds thereof are therefore intended to be embracedby the claims.

1. On a computing system comprising a multi-touch display, a method ofmulti-modal interaction, the method comprising: recognizing a handposture of a user's first hand directed at the display; displaying onthe display a modal region based on the hand posture, the modal regiondefining an area on the display; applying a mode to the modal region,the mode indicating a family of functionalities to be associated withthe modal region and defining a mapping of one or more touch gestures toone or more corresponding actions associated with the mode; while themodal region remains displayed, recognizing a touch gesture from auser's second hand directed at the display within the modal region onthe display; and performing an action on the display based upon amapping of the touch gesture from the user's second hand to the action.2. The method of claim 1, further comprising displaying the modal regionat a location on the display based on a location of the hand posture. 3.The method of claim 2, further comprising detecting a change in thelocation of the hand posture and in response, adjusting the location ofthe modal region.
 4. The method of claim 1, further comprising detectinga hand posture of a second user's hand directed at the display, and inresponse, displaying on the display a second modal region.
 5. The methodof claim 4, wherein the hand posture of the second user's hand and thehand posture of the user's first hand are temporally overlapping.
 6. Themethod of claim 1, further comprising detecting lifting of the user'sfirst hand, and in response, continuing to display the modal regionafter lifting is detected.
 7. The method of claim 1, further comprisingdetecting lifting of the user's first hand, and in response, ceasingdisplay of the modal region.
 8. The method of claim 1, furthercomprising receiving a locking input, detecting lifting of the user'sfirst hand after receiving the locking input, and continuing display ofthe modal region after the user's first hand is lifted.
 9. The method ofclaim 1, wherein applying the mode comprises applying a default mode tothe modal region.
 10. The method of claim 1, wherein applying the modeincludes receiving an input selecting the mode via a touch inputdirected at a virtual control displayed on the display.
 11. In a displaydevice comprising a multi-touch display, a computing device comprising:a processor; and memory comprising instructions stored thereon that areexecutable by the processor to: recognize a hand posture of a user'sfirst hand touching the display; display on the display a modal regionbased on the hand posture, the modal region defining an area on thedisplay and being displayed at a location on the display based on alocation of the hand posture; receive an input selecting a mode to beapplied to the modal region, the mode indicating a family offunctionalities to be associated with the modal region and defining amapping of one or more touch gestures to one or more correspondingactions associated with the mode; detect a change in the location of thehand posture and in response, adjust the location of the modal region;while the modal region remains displayed, recognize a touch gesture froma second hand directed at the display within the modal region; performan action on the display based upon a mapping of the touch gesture fromthe second hand to the action; and recognize a removal of the handposture from the display and in response, remove display of the modalregion from the display.
 12. The computing device of claim 11, whereinthe instructions are further executable to determine that the modalregion is to remain displayed if the hand posture of the user's firsthand is still touching the display device.
 13. The computing device ofclaim 11, wherein the instructions are further executable to recognize aplurality of hand postures, and to display on the display a second modalregion associated with each hand posture recognized.
 14. The computingdevice of claim 13, wherein the modal region is a first modal region,and wherein the instructions are further executable to detect apotential overlap of the first modal region and the second modal regiondisplayed on the display, to determine whether displaying the firstmodal region and the second modal region as overlapping is allowed orunallowed based upon a comparison of the mode applied to the first modalregion and a mode applied to the second modal region, and to display thefirst modal region and the second modal region as overlapping or notoverlapping depending upon the comparison.
 15. The computing device ofclaim 11, wherein the instructions are executable to perform the actionon the display by changing a property of content displayed on thedisplay within the modal region.
 16. The computing device of claim 11,wherein the instructions are executable to perform the action on thedisplay by changing a property of content displayed on the displayoutside of the modal region
 17. In a computing system comprising amulti-touch display, a method of interacting with a plurality of users,the method comprising: recognizing a hand posture of a first hand of afirst user directed at the display; in response to recognizing the handposture of the first hand of the first user, displaying on the display afirst modal region based on the hand posture of the first hand of thefirst user, the first modal region defining an area on the display;receiving an input selecting a mode to be applied to the first modalregion, the mode indicating a family of functionalities to be associatedwith the first modal region and defining a mapping of one or more touchgestures to one or more corresponding actions associated with the mode;while the first modal region remains displayed, recognizing a touchgesture from a second hand of the first user directed at the displaywithin the first modal region, and in response, performing a firstaction on the display based upon a mapping of the touch gesture from thesecond hand of the first user to the first action; while the first modalregion remains displayed, recognizing a hand posture of a first hand ofa second user directed at the display; in response to recognizing thehand posture of the first hand of the second user, displaying on thedisplay a second modal region; receiving an input selecting a mode to beapplied to the second modal region; and while the second modal regionremains displayed, recognizing a touch gesture from a second hand of thesecond user directed at the display within the second modal region, andin response, performing a second action on the display based upon amapping of the touch gesture from the second hand of the second user tothe second action.
 18. The method of claim 17, wherein the mode appliedto the first modal region and the mode applied to the second modalregion are a same mode.
 19. The method of claim 17, wherein the modeapplied to the first modal region and the mode applied to the secondmodal region are different modes.
 20. The method of claim 17, furthercomprising displaying the first modal region at a location on thedisplay based on a location of the hand posture of the first hand of thefirst user, and displaying the second modal region at a location on thedisplay based on a location of the hand posture of the first hand of thesecond user.